Skip to content

Add leader election to controller manager #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 10, 2018

Conversation

JoelSpeed
Copy link
Contributor

Fixes #93

This is a first pass at implementing leader election code into the manager.

I've created a leaderelection package which contains constructors for a resourcelock.Interface as well as a fake for use in testing.

I've added three new options to the manager options for leader election, a bool to enable the feature (disabled by default to preserve behaviour) and a name and namespace for the configmap that it will use for the lock.

I was considering factoring out the Start test suite such that you can run it with different Options, since that's all I do in the tests for the leader election and theres a lot of duplication of the test suite. Does that seem like a reasonable thing to do?

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 24, 2018
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 24, 2018
@DirectXMan12
Copy link
Contributor

I was considering factoring out ... Does that seem like a reasonable thing to do?

Yeah, it's probably a decent idea.

Copy link
Contributor

@DirectXMan12 DirectXMan12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is configmap-based leader election the reccomended default these days? I'm fine with us choosing an opinionated option, but I want to make sure it's the right one, and we should document that.

Also, we should choose a sane default for the configmap id (and potentially namespace) if we can.

@JoelSpeed
Copy link
Contributor Author

I have refactored the test suite to reduce the duplication.

I asked in Slack in #SIG-api-machinery and got confirmation that configmaps are preferrred over endpoints, however in 1.12 there will be a dedicated object coming for this purpose, so perhaps make a TODO for upgrading it once the controller-runtime gets upgraded to 1.12.

Where would you default the ID and the namespace? I imagined these would be defaulted by adding some flags to wherever the manager is initialised (one of the templates in controller-tools), mine looks like this at the moment:

	mgr, err := manager.New(cfg, manager.Options{
		LeaderElection:          *leaderElection,
		LeaderElectionID:        *leaderElectionID,
		LeaderElectionNamespace: *leaederElectionNamespace,
	})

Where those variables come from flags.

@DirectXMan12
Copy link
Contributor

TODO for upgrading it once the controller-runtime gets upgraded to 1.12.

Yes, please add the TODO (// TODO(JoelSpeed): switch to leaderelection object in 1.12) and/or file an issue

Where would you default the ID and the namespace

Namespace should default to the current namespace, when we can figure out it (e.g. via some default downward API environment variable). ID should have some sane default assuming this is the only controller in the namespace, and then we can make it overridable if need be. We should endeavor to make it so that the defaults work when using the setups that we describe in our docs and in kubebuilder.

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 29, 2018
@JoelSpeed
Copy link
Contributor Author

Yes, please add the TODO (// TODO(JoelSpeed): switch to leaderelection object in 1.12) and/or file an issue

Added.

Namespace should default to the current namespace

I've added a getInClusterNamespace that attempts to read the namespace from the service account credentials added to a Pod. This should work in most cases I believe. If people aren't mounting service account credentials for their controllers then this won't work however, I can't think when that would be the case though. It returns a "nice" error suggesting you should set the namespace when it isn't running in cluster.

ID should have some sane default assuming this is the only controller in the namespace

I've defaulted it to controller-runtime for now, is about the most sensible thing I could come up with.

@DirectXMan12
Copy link
Contributor

I like the defaulting logic, I think this looks decent. Does the leader election ConfigMap get created with the ID as the name, verbatim? If so, perhaps something like "controller-leader-election-helper" for the ID.

🚲 🏠

@JoelSpeed
Copy link
Contributor Author

Does the leader election ConfigMap get created with the ID as the name, verbatim?

Yes, precisely. If the ID is controller-runtime then the ConfigMap would be called controller-runtime, I'm happy to change it as per your suggestion, I'll make the change

return nil, err
}

// TODO(JoelSpeed): switch to leaderelection object in 1.12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious to know more about this but am having a hard time finding other references to it. Do you have any links to info about this object?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I asked about it in #sig-api-machinery, and got the info from there: https://kubernetes.slack.com/archives/C0EG7JC6T/p1535318981000100

I know nothing more about it, sorry

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JoelSpeed
Copy link
Contributor Author

There were merge conflicts so I've just rebased on top of master

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 3, 2018
@JoelSpeed
Copy link
Contributor Author

I fixed the tests that were failing, I think this might be ready to go?

PTAL @DirectXMan12 @pwittrock

@DirectXMan12
Copy link
Contributor

Yeah, sorry, trying to do one last scan-through before lgtm.

@DirectXMan12
Copy link
Contributor

To be clear, when we lose the leader election lock, we exit out, and presumably the pod gets restarted? Can you add a note in the cm.start or Start logic about that? Otherwise, it's a bit confusing to reason through. Afterwards, good to go.

@JoelSpeed
Copy link
Contributor Author

@DirectXMan12 I've added a note about the behaviour and what to expect when a leader loses leadership. Does that seem sufficient?

@DirectXMan12
Copy link
Contributor

Perfect. Can you squash down into "logical" commits? We can merge after that.

@JoelSpeed
Copy link
Contributor Author

@DirectXMan12 Squashed as per your request 😄

@DirectXMan12
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 10, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DirectXMan12, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 10, 2018
@k8s-ci-robot k8s-ci-robot merged commit 419794c into kubernetes-sigs:master Sep 10, 2018
justinsb pushed a commit to justinsb/controller-runtime that referenced this pull request Dec 7, 2018
Add leader election to controller manager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants